Overview of the CLEF 2005 Multilingual Question Answering Track
Identifieur interne : 000616 ( Main/Exploration ); précédent : 000615; suivant : 000617Overview of the CLEF 2005 Multilingual Question Answering Track
Auteurs : Alessandro Vallin [Italie] ; Bernardo Magnini [Italie] ; Danilo Giampiccolo [Italie] ; Lili Aunimo [Finlande] ; Christelle Ayache [France] ; Petya Osenova [Bulgarie] ; Anselmo Pe As [Espagne] ; Maarten De Rijke [Pays-Bas] ; Bogdan Sacaleanu [Allemagne] ; Diana Santos [Norvège] ; Richard Sutcliffe [Irlande (pays)]Source :
- Lecture Notes in Computer Science [ 0302-9743 ]
Abstract
Abstract: The general aim of the third CLEF Multilingual Question Answering Track was to set up a common and replicable evaluation framework to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages. Nine target languages and ten source languages were exploited to enact 8 monolingual and 73 cross-language tasks. Twenty-four groups participated in the exercise. Overall results showed a general increase in performance in comparison to last year. The best performing monolingual system irrespective of target language answered 64.5% of the questions correctly (in the monolingual Portuguese task), while the average of the best performances for each target language was 42.6%. The cross-language step instead entailed a considerable drop in performance. In addition to accuracy, the organisers also measured the relation between the correctness of an answer and a system’s stated confidence in it, showing that the best systems did not always provide the most reliable confidence score. We provide an overview of the 2005 QA track, detail the procedure followed to build the test sets and present a general analysis of the results.
Url:
DOI: 10.1007/11878773_36
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 002732
- to stream Istex, to step Curation: 001E94
- to stream Istex, to step Checkpoint: 000563
- to stream Main, to step Merge: 000621
- to stream Main, to step Curation: 000616
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Overview of the CLEF 2005 Multilingual Question Answering Track</title><author><name sortKey="Vallin, Alessandro" sort="Vallin, Alessandro" uniqKey="Vallin A" first="Alessandro" last="Vallin">Alessandro Vallin</name></author><author><name sortKey="Magnini, Bernardo" sort="Magnini, Bernardo" uniqKey="Magnini B" first="Bernardo" last="Magnini">Bernardo Magnini</name></author><author><name sortKey="Giampiccolo, Danilo" sort="Giampiccolo, Danilo" uniqKey="Giampiccolo D" first="Danilo" last="Giampiccolo">Danilo Giampiccolo</name></author><author><name sortKey="Aunimo, Lili" sort="Aunimo, Lili" uniqKey="Aunimo L" first="Lili" last="Aunimo">Lili Aunimo</name></author><author><name sortKey="Ayache, Christelle" sort="Ayache, Christelle" uniqKey="Ayache C" first="Christelle" last="Ayache">Christelle Ayache</name></author><author><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name></author><author><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></author><author><name sortKey="De Rijke, Maarten" sort="De Rijke, Maarten" uniqKey="De Rijke M" first="Maarten" last="De Rijke">Maarten De Rijke</name></author><author><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></author><author><name sortKey="Santos, Diana" sort="Santos, Diana" uniqKey="Santos D" first="Diana" last="Santos">Diana Santos</name></author><author><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:994CAB05CE3090F1FFFFB5D04BE83629F9F1A40B</idno><date when="2006" year="2006">2006</date><idno type="doi">10.1007/11878773_36</idno><idno type="url">https://api.istex.fr/ark:/67375/HCB-39C220BM-P/fulltext.pdf</idno><idno type="wicri:Area/Istex/Corpus">002732</idno><idno type="wicri:explorRef" wicri:stream="Istex" wicri:step="Corpus" wicri:corpus="ISTEX">002732</idno><idno type="wicri:Area/Istex/Curation">001E94</idno><idno type="wicri:Area/Istex/Checkpoint">000563</idno><idno type="wicri:explorRef" wicri:stream="Istex" wicri:step="Checkpoint">000563</idno><idno type="wicri:doubleKey">0302-9743:2006:Vallin A:overview:of:the</idno><idno type="wicri:Area/Main/Merge">000621</idno><idno type="wicri:Area/Main/Curation">000616</idno><idno type="wicri:Area/Main/Exploration">000616</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Overview of the CLEF 2005 Multilingual Question Answering Track</title><author><name sortKey="Vallin, Alessandro" sort="Vallin, Alessandro" uniqKey="Vallin A" first="Alessandro" last="Vallin">Alessandro Vallin</name><affiliation wicri:level="1"><country xml:lang="fr">Italie</country><wicri:regionArea>CELCT</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Italie</country></affiliation></author><author><name sortKey="Magnini, Bernardo" sort="Magnini, Bernardo" uniqKey="Magnini B" first="Bernardo" last="Magnini">Bernardo Magnini</name><affiliation wicri:level="1"><country xml:lang="fr">Italie</country><wicri:regionArea>ITC-Irst</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Italie</country></affiliation></author><author><name sortKey="Giampiccolo, Danilo" sort="Giampiccolo, Danilo" uniqKey="Giampiccolo D" first="Danilo" last="Giampiccolo">Danilo Giampiccolo</name><affiliation wicri:level="1"><country xml:lang="fr">Italie</country><wicri:regionArea>CELCT</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Italie</country></affiliation></author><author><name sortKey="Aunimo, Lili" sort="Aunimo, Lili" uniqKey="Aunimo L" first="Lili" last="Aunimo">Lili Aunimo</name><affiliation wicri:level="1"><country xml:lang="fr">Finlande</country><wicri:regionArea>University of Helsinki</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Finlande</country></affiliation></author><author><name sortKey="Ayache, Christelle" sort="Ayache, Christelle" uniqKey="Ayache C" first="Christelle" last="Ayache">Christelle Ayache</name><affiliation wicri:level="1"><country xml:lang="fr">France</country><wicri:regionArea>ELDA/ELRA</wicri:regionArea></affiliation><affiliation></affiliation></author><author><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name><affiliation wicri:level="1"><country xml:lang="fr">Bulgarie</country><wicri:regionArea>BTB</wicri:regionArea></affiliation><affiliation></affiliation></author><author><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name><affiliation wicri:level="1"><country xml:lang="fr">Espagne</country><wicri:regionArea>UNED</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Espagne</country></affiliation></author><author><name sortKey="De Rijke, Maarten" sort="De Rijke, Maarten" uniqKey="De Rijke M" first="Maarten" last="De Rijke">Maarten De Rijke</name><affiliation wicri:level="1"><country xml:lang="fr">Pays-Bas</country><wicri:regionArea>University of Amsterdam</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Pays-Bas</country></affiliation></author><author><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name><affiliation wicri:level="1"><country xml:lang="fr">Allemagne</country><wicri:regionArea>DFKI</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Allemagne</country></affiliation></author><author><name sortKey="Santos, Diana" sort="Santos, Diana" uniqKey="Santos D" first="Diana" last="Santos">Diana Santos</name><affiliation wicri:level="1"><country xml:lang="fr">Norvège</country><wicri:regionArea>Sintef</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Norvège</country></affiliation></author><author><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name><affiliation wicri:level="1"><country xml:lang="fr">Irlande (pays)</country><wicri:regionArea>University of Limerick</wicri:regionArea></affiliation><affiliation wicri:level="1"><country wicri:rule="url">Irlande (pays)</country></affiliation></author></analytic><monogr></monogr><series><title level="s" type="main" xml:lang="en">Lecture Notes in Computer Science</title><idno type="ISSN">0302-9743</idno><idno type="eISSN">1611-3349</idno><idno type="ISSN">0302-9743</idno></series></biblStruct></sourceDesc><seriesStmt><idno type="ISSN">0302-9743</idno></seriesStmt></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: The general aim of the third CLEF Multilingual Question Answering Track was to set up a common and replicable evaluation framework to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages. Nine target languages and ten source languages were exploited to enact 8 monolingual and 73 cross-language tasks. Twenty-four groups participated in the exercise. Overall results showed a general increase in performance in comparison to last year. The best performing monolingual system irrespective of target language answered 64.5% of the questions correctly (in the monolingual Portuguese task), while the average of the best performances for each target language was 42.6%. The cross-language step instead entailed a considerable drop in performance. In addition to accuracy, the organisers also measured the relation between the correctness of an answer and a system’s stated confidence in it, showing that the best systems did not always provide the most reliable confidence score. We provide an overview of the 2005 QA track, detail the procedure followed to build the test sets and present a general analysis of the results.</div></front></TEI><affiliations><list><country><li>Allemagne</li><li>Bulgarie</li><li>Espagne</li><li>Finlande</li><li>France</li><li>Irlande (pays)</li><li>Italie</li><li>Norvège</li><li>Pays-Bas</li></country></list><tree><country name="Italie"><noRegion><name sortKey="Vallin, Alessandro" sort="Vallin, Alessandro" uniqKey="Vallin A" first="Alessandro" last="Vallin">Alessandro Vallin</name></noRegion><name sortKey="Giampiccolo, Danilo" sort="Giampiccolo, Danilo" uniqKey="Giampiccolo D" first="Danilo" last="Giampiccolo">Danilo Giampiccolo</name><name sortKey="Giampiccolo, Danilo" sort="Giampiccolo, Danilo" uniqKey="Giampiccolo D" first="Danilo" last="Giampiccolo">Danilo Giampiccolo</name><name sortKey="Magnini, Bernardo" sort="Magnini, Bernardo" uniqKey="Magnini B" first="Bernardo" last="Magnini">Bernardo Magnini</name><name sortKey="Magnini, Bernardo" sort="Magnini, Bernardo" uniqKey="Magnini B" first="Bernardo" last="Magnini">Bernardo Magnini</name><name sortKey="Vallin, Alessandro" sort="Vallin, Alessandro" uniqKey="Vallin A" first="Alessandro" last="Vallin">Alessandro Vallin</name></country><country name="Finlande"><noRegion><name sortKey="Aunimo, Lili" sort="Aunimo, Lili" uniqKey="Aunimo L" first="Lili" last="Aunimo">Lili Aunimo</name></noRegion><name sortKey="Aunimo, Lili" sort="Aunimo, Lili" uniqKey="Aunimo L" first="Lili" last="Aunimo">Lili Aunimo</name></country><country name="France"><noRegion><name sortKey="Ayache, Christelle" sort="Ayache, Christelle" uniqKey="Ayache C" first="Christelle" last="Ayache">Christelle Ayache</name></noRegion></country><country name="Bulgarie"><noRegion><name sortKey="Osenova, Petya" sort="Osenova, Petya" uniqKey="Osenova P" first="Petya" last="Osenova">Petya Osenova</name></noRegion></country><country name="Espagne"><noRegion><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></noRegion><name sortKey="Pe As, Anselmo" sort="Pe As, Anselmo" uniqKey="Pe As A" first="Anselmo" last="Pe As">Anselmo Pe As</name></country><country name="Pays-Bas"><noRegion><name sortKey="De Rijke, Maarten" sort="De Rijke, Maarten" uniqKey="De Rijke M" first="Maarten" last="De Rijke">Maarten De Rijke</name></noRegion><name sortKey="De Rijke, Maarten" sort="De Rijke, Maarten" uniqKey="De Rijke M" first="Maarten" last="De Rijke">Maarten De Rijke</name></country><country name="Allemagne"><noRegion><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></noRegion><name sortKey="Sacaleanu, Bogdan" sort="Sacaleanu, Bogdan" uniqKey="Sacaleanu B" first="Bogdan" last="Sacaleanu">Bogdan Sacaleanu</name></country><country name="Norvège"><noRegion><name sortKey="Santos, Diana" sort="Santos, Diana" uniqKey="Santos D" first="Diana" last="Santos">Diana Santos</name></noRegion><name sortKey="Santos, Diana" sort="Santos, Diana" uniqKey="Santos D" first="Diana" last="Santos">Diana Santos</name></country><country name="Irlande (pays)"><noRegion><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></noRegion><name sortKey="Sutcliffe, Richard" sort="Sutcliffe, Richard" uniqKey="Sutcliffe R" first="Richard" last="Sutcliffe">Richard Sutcliffe</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Informatique/explor/SgmlV1/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000616 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000616 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Informatique
|area= SgmlV1
|flux= Main
|étape= Exploration
|type= RBID
|clé= ISTEX:994CAB05CE3090F1FFFFB5D04BE83629F9F1A40B
|texte= Overview of the CLEF 2005 Multilingual Question Answering Track
}}
| This area was generated with Dilib version V0.6.33. |  |